Systems, methods, and apparatuses for processing video

ABSTRACT

Systems, methods, and apparatuses are described for processing video. Video content comprising a plurality of frames may be received. A viewing parameters associated with playback of the video content may be determined. One or more of luminance pixel data associated with a frame of the plurality of frames and chrominance pixel data associated with the frame is determined. A quantization matrix associated with the frame may be determined based on the viewing parameter and the one or more of luminance pixel data and chrominance pixel data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/376,242, filed Apr. 5, 2019, which claims priority to U.S.Provisional Application No. 62/653,891, filed Apr. 6, 2018, each ofwhich is incorporated by reference herein in its entirety.

BACKGROUND

Video and/or images often present highly bright as well as very darksignals on the same video frame or image, thereby providing a highcontrast ratio within the same image. However, coding (e.g.,compression) of the video or image content remains a challenge due touser demand for high visual quality, which in turn requires allocatingmore bits and significantly increasing video coding depth to achieve thedesired user quality. In addition, the transmission bandwidth is limiteddue to typical limitations of the existing network infrastructures,especially when transmitting over wireless/cellular networks. As such,improvements are needed.

SUMMARY

Systems, methods, and apparatuses are described for processing video.Raw (i.e., uncompressed) video data of video content may be received anda portion (e.g., a partition) of a frame of the raw video data may bedesignated for encoding. The partition of the frame may be with respectto luminance pixel data, chrominance pixel data, or both. The image dataof the partition may be subject to quantization during an encodingprocess. The quantization may comprise application of a quantizationmatrix. The quantization matrix may be based on one or more viewingparameters associated with playback of the video content. Such aquantized partition may omit image data at some spatial frequencies thatwould not be or are expected not to be perceivable by a viewer underthose viewing parameters. The quantized partition may be incorporatedinto an encoded video stream that is received by a decoder.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings show generally, by way of example, but not by wayof limitation, various examples discussed in the present disclosure. Inthe drawings:

FIG. 1 is a block diagram of a video distribution system;

FIG. 2 is a graphical representation of values relating to contrastsensitivity;

FIG. 3 is a flow diagram of a method;

FIG. 4 is a flow diagram of a method;

FIG. 5 is a flow diagram of a method; and

FIG. 6 is a block diagram of a system environment.

DETAILED DESCRIPTION

Systems, methods, and apparatuses are described for processing (e.g.,preprocessing, coding, pre-filtering, partitioning, etc.) video data.Video (e.g., High Dynamic Range (HDR) video and other types of video)often comprises relatively high luminance levels and fine shadowdetails, which extend far beyond some conventional types of video (e.g.,Standard Dynamic Range (SDR) video). Video, such as the noted HDR video,may comprise highly bright as well as very dark signals on the samevideo frame, thereby providing a high contrast ratio within the sameimage. For example, HDR has gained popularity with the finalization ofthe High Efficiency Video Coding (HEVC) standard, which allowscompressing high-resolution HDR content, such as 3840×2160 (4K)resolutions in terms of luma samples, in a more efficient mannercompared to its predecessor H.264/Moving Picture Experts Group(MPEG)-Advanced Video Coding (AVC).

In some cases, visually perceived coding artifacts may beunintentionally introduced into video content during processing (e.g.,encoding, compression, etc.). For example, a certain degree ofcompression may be required to keep compressed video within particularbandwidth constraints (e.g., 6 Megabits (Mb) to 14 Mb per second),thereby introducing said coding artifacts. Moreover, encoding video,such as HDR video, may consume significant computational resources, suchas to preserve fine details within the video. Therefore, there is a needto improve the perceived visual quality of the compressed video withoutincreasing its bit-rate.

A viewer may be unable to perceive certain details of a video or imagethereof, such as due to limitations of the human eye (which may berepresented by the human visual system (HVS)). A viewer may be unable todifferentiate between details in the video/image having a high spatialfrequency. A viewer may be unable to perceive very minor details in animage and/or very minor movements over several images of the video. Anumber of factors (“viewing parameters”) relating to the viewingexperience may influence the viewer's ability to perceive certain videodata. For example, one or more of a viewing distance, a pixel density, apixel length of a sinusoidal grating cycle, a width of a displayassociated with playback of the video content, a viewing angle, ambientillumination level, and a reflection coefficient of a display associatedwith playback of the video content may influence the viewer's ability toperceive video data.

In video codecs, quantization matrices may be applied in a frequencydomain to transformed pixels (or residuals). Each transform coefficientmay be divided by a value defined in the quantization matrix. Valuescorresponding to high frequencies may be assigned elements within thequantization matrix to effectively zero out the high frequency values.

Encoding/bandwidth/resource cost for areas of a content asset (e.g.,video content) that a user cannot perceive may be reduced, such as byuse of perceptual quantization matrices to encode frames (or portions offrames) in a particular manner. Based on factors for defining theperceptual quantization matrices and applying contrast sensitivityfunctions (CSFs) to each entry in the perceptual quantization matrices,the encoder may more efficiently allocate resources to frames orblocks/portions of frames that a user is more likely to perceive. Incontrast, blocks/portions of frames that the user is less likely toperceive may be removed and/or be devoted less processing resources.

Viewing parameters associated with the video playback may be leveragedto determine which portions of video data may be omitted in thecompressed video data while still maintaining at least the same level ofperceived video quality. A quantization matrix that is applied duringthe video encoding process (or preprocessing) may be determined based onone or more viewing parameters. A CSF may be determined based on theviewing parameter(s) and that, in turn, may be used to determine thequantization matrix.

The size of the compressed video may be reduced, thus saving on storageand bandwidth. Coding of those portions of the video that areperceivable to a viewer may be improved. The processing resources and/orbits that would have been devoted to the omitted video data may beinstead applied to the perceivable portions of the video.

FIG. 1 shows a block diagram of a video distribution system 100. Thevideo distribution system 100 may comprise a video source 110, a videopreprocessor 120, a video encoder 130, and a video decoder 140. Thevideo source 110 may transmit (e.g., send, deliver, provide, etc.) raw,uncompressed video data, comprising video content (e.g., audio and/orvisual), to the video preprocessor 120 and/or the video encoder 130. Thevideo preprocessor 120 may process the raw video data and transmit(e.g., send, deliver, provide, etc.) the preprocessed video data to thevideo encoder 130. The video encoder 130 may receive the raw video dataand/or the preprocessed video data, as the case may be, and encode suchvideo data. The encoded video data may be transmitted (e.g., sent,delivered, provided, etc.) to the video decoder 140. The video decoder140 may decode the encoded video data. The decoded video data,comprising the video content, may be presented (e.g., caused to beoutput) to a viewer.

The video source 110 may transmit (e.g., send, deliver, provide, etc.)video (audio and/or visual) content based on a request. The request maybe from a user device, such as a mobile device, a set-top box, a cablemodem, etc. The request may be based on a schedule (such as a network'sprogramming schedule), which automatically requests video contentrelated to a particular program at a particular time. The video contentmay comprise a plurality of frames, a single frame among a plurality offrames, or a single independent frame (e.g., not part of a plurality ofvideo frames (ordered or otherwise)). Video content may additionally oralternatively comprise associated audio content. The video source 110may be video storage, such as from a video-on-demand database. The videosource 110 may be a video stream, such as a live feed from a camera. Thevideo content transmitted (e.g., sent, delivered, provided, etc.) fromthe video source 110 may be raw (e.g., uncompressed, unprocessed, notencoded, etc.) video.

The video preprocessor 120 may receive (e.g., intercept, etc.) videodata from the video source 110. The video preprocessor 120 may compriseone or more logical blocks of instructions 122, 124 for preprocessingvideo. The video source 110 and the video preprocessor 120 may beintegrated into one or more computing devices. The video source 110 andthe video preprocessor 120 may be local to one another (e.g., in thesame room, on the same premises, etc.). The video source 110 and thevideo preprocessor 120 may be remote from one another. The videopreprocessor 120 may execute one or more of the one or more logicalblocks of instructions 122, 124 to convert received raw video intopreprocessed (e.g., coded, etc.) video.

The video preprocessor 120 and/or the video source 110 may transmit(e.g., send, deliver, provide, etc.) the video data (raw orpreprocessed) to the video encoder 130. The video encoder 130 maycomprise one or more logical blocks of instructions 132, 134 forencoding video data. The video encoder 130 may treat the received videodata as input. The video encoder 130 may execute one or more of the oneor more logical blocks of instructions 132, 134 to convert received rawvideo data and/or preprocessed video data into encoded (e.g.,compressed, etc.) video data.

The video encoder 130 may encode the video data based on one or more ofa variety of video coding formats, such as MPEG-2 Part 2, MPEG-4 Part 2,H.264 (MPEG-4 Part 10), or H.265 (HEVC). The video data may be organizedas a series of frames. The series of frames may comprise I-, B-, andP-frames. Each frame may be divided into a number of partitions. Eachpartition may comprise a plurality of pixels. Depending on the codingformat, the partition may be a block, macroblock, coding tree unit, etc.A partition may comprise an 8×8 block of image values. A partition maycomprise any other size block of image values. An image value of apartition may represent a pixel. A partition may be with respect to luma(Y) or one of the two chroma components (Cr (U), Cb (V)). The imagevalues of a partition may indicate a luminance component or one of thetwo chrominance components of the respective pixel of the partition.

The video encoder 130 may convert the image values to the spatialfrequency domain using a linear transform, such as a discrete cosinetransform (DCT). The DCT may transform a matrix of image values to acorresponding matrix of frequency transform coefficients. The transformcoefficients may enable the initial matrix of image values to berecreated. The transform coefficients each may reflect the relativeweighting of the respective spatial frequency to be used to re-transformthe partition to its original image values. The transform coefficientsmay indicate the relative contribution or impact that the respectivespatial frequency provides in defining the image represented in thepartition.

The transform coefficient entries of the DCT-transformed matrix thatrepresent the lower spatial frequencies may be higher values, while thetransform coefficient entries that represent the higher spatialfrequencies may be lower values, often much lower. Since the higherspatial frequencies may be less noticeable to the human eye—and,therefore, provide relatively little value in maintaining perceivedimage quality—image data at the higher spatial frequencies may beomitted or ignored when encoding the partition. The determination of thespatial frequencies to omit may be based, at least in part, on viewingparameters relating to playback (e.g., anticipated and/or actual) of thevideo content.

The video encoder 130 may determine which spatial frequencies torepresent in the encoded video during a quantization step. Quantizationmay effectively downscale the spatial-frequency domain of theDCT-transformed matrix. Quantization of the DCT-transformed matrix maybe based on the quantization matrix. The quantization matrix may havedimensions equal to those of the DCT-transformed matrix. The quantizedDCT-transformed matrix, determined based on the quantization matrix, maycomprise one or more quantized transform coefficients having a zero (0)value. The image data at spatial frequencies corresponding to thequantized transform coefficients having a 0 value may be omitted fromthe encoded video data.

The quantization matrix may be associated with intra-coding and/orinter-coding. A first quantization matrix may be associated withintra-coding and a second quantization matrix may be associated withinter-coding. The first quantization matrix may be applied tointra-predicted blocks (e.g., intra-predicted partitions) and the secondquantization matrix may be applied to inter-predicted blocks (e.g.,inter-predicted partitions). A quantization matrix for aninter-predicted block may be based on a quantization matrix for anassociated intra-predicted block. A quantization matrix for anintra-predicted block may be based on a quantization matrix for anassociated inter-predicted block. The inter-predicted block and theintra-predicted block may be blocks of the same frame.

The quantization matrix may be associated with a luma component and/or achroma component. The chroma component may comprise a first chromacomponent and a second chroma component. Different quantization matricesmay be applied to a luma component and a corresponding chroma componentof the same partition. A first quantization matrix may be associatedwith a luma component, a second quantization matrix may be associatedwith a first chroma component, and a third quantization matrix may beassociated with a second chroma component. The first quantization matrixmay be applied to a luma component of a partition, the secondquantization matrix may be applied to a first chroma component of thepartition, and the third quantization matrix may be applied to a secondchroma component of the partition.

The quantization matrix may be used to determine, based on known orestimated viewing parameter(s), the spatial frequencies at which imagedata is omitted from the encoded video data and the spatial frequenciesat which image data is included in the encoded video data. Thus, thequantization matrix may be determined based on one or more of theviewing parameters associated with playback (or expected playback) ofthe video data. The viewing parameter may comprise one or more of aviewing distance, a pixel density, a pixel length of a sinusoidalgrating cycle, a width of a display associated with playback of thevideo content, a viewing angle, ambient illumination level, and areflection coefficient of a display associated with playback of thevideo content.

The determination of a quantization matrix may not be limited to merelyincreasing or decreasing a universal spatial frequency threshold for thepartition. The quantization matrix may be determined to affecthorizontal spatial frequency and vertical spatial frequency independentof one another. A quantization matrix may be determined that tends tocause image data with high vertical spatial frequency to be omitted butallows image data with high horizontal spatial frequency to be retainedin the encoded video data, and vice versa. The quantization matrix maybe determined to achieve any combination of possible spatial frequencyprofiles.

The quantization matrix may be determined based on a CSF. The CSF may bebased on the one or more viewing parameters. A CSF may be definedaccording to the relationship between a variable spatial frequency and avariable contrast sensitivity (e.g., defined by the HVS). A CSF mayrepresent the threshold spatial frequency and/or contrast sensitivity atwhich the image data at that spatial frequency is not perceptible by aviewer. In particular, the determined CSF may represent the thresholdspatial frequency and/or contrast sensitivity at which the image data atthat spatial frequency is not perceptible by a viewer under viewingconditions indicated as the viewing parameters.

The quantization matrix for a partition may be based on a characteristicof the partition. The quantization matrix for a partition may be basedon a characteristic of the frame comprising the partition. Acharacteristic of the partition and/or the frame may be the videocontent indicated by the partition and/or frame. The video content maycomprise visual details (with respect to luma and/or chroma) having ahigh spatial frequency (e.g., a spatial frequency above a thresholdspatial frequency). The visual details having a high spatial frequencymay indicate small details in the video content. The video content maycomprise a texture. A characteristic of the partition and/or frame maybe motion activity. The motion activity may be observed in the videocontent indicated by the partition and/or the frame. The motion activitymay be indicated by one or more motion vectors associated with thepartition and/or frame. A characteristic of the partition and/or framemay be a resolution of the partition and/or the frame. A characteristicof the partition and/or frame may be a resolution of the video contentindicated by the partition and/or frame.

A characteristic of the partition and/or frame may be a quadtreestructure associated with the partition and/or frame. A characteristicof the partition and/or frame may be a residual quadtree structureassociated with the partition and/or frame. A characteristic of thepartition and/or frame may be the size (e.g. the dimensions) of thepartition and/or frame. A characteristic of the partition and/or framemay be a size of a coding unit associated with the partition and/orframe. A characteristic of the partition and/or frame may be a size ofone or more transform units associated with the partition and/or frame.

The quantization matrix may be based on the sequential position of theframe within the video stream. Different quantization matrices may bedetermined for different portions of a video stream. The quantizationmatrix may be based on the group of pictures (GOP) to which the frame isa part. The quantization matrix may be based on the frame's relation toa scene cut. The frame's relation to the scene cut may be with respectto the display time of the frame and the display time of the scene cut.The frame's relation to the scene cut may be with respect to thesequential difference, in the display time ordering, between the frameand the scene cut frame(s).

Determining the quantization matrix based on the viewing parametersand/or the CSF may comprise selecting the quantization matrix from aplurality of candidate quantization matrices. The candidate quantizationmatrices may each correspond to various profiles of viewing parameters.The quantization matrix may be based on the candidate viewing parameterhaving a viewing parameter profile that most closely matches the actualviewing parameters of the viewing environment.

The quantization matrix may be realized in different dimensions, such as4×4, 8×8, 16×16, or 32×32. The transform matrix or matrices, to whichthe quantization matrix may be applied, may be sized in correspondingdimensions as the quantization matrix. A quantization matrix dimensionedin a relatively small size (e.g., 4×4 or 8×8) may be up-sampled todetermine a larger quantization matrix (e.g., 32×32).

The video encoder 130 may use a prediction function as part of theencoding process. The prediction may be performed with respect to apartition of a frame. The prediction may be based on the same frame(intra prediction), another frame (inter prediction), or both. Theprediction may be spatial or temporal. The prediction may comprisemotion estimation, which may also be spatial or temporal. The imagevalues of a partition may indicate a residual value between the rawimage values and those resulting from the partition. The residual valuesmay be subjected to transform, quantization, etc.

The video encoder 130 may cause the matrix of quantized transformcoefficients for the partition to undergo entropy encoding (e.g.,Huffman coding, etc.), with the output being added to the coded videostream. The entropy encoding may be a lossless compression. The entropyencoding may be a run-length encoding algorithm. In the entropy encodingprocess, the redundant quantized transform coefficients may becompressed into a representation requiring less bits than wouldotherwise be the case.

Although the systems, methods, and apparatuses described herein aregenerally described in reference to the video encoder 130, it isunderstood that the video preprocessor 120 may perform, at apreprocessing stage, some or all of the disclosed techniques forremoving image data occurring at imperceptible spatial frequencies basedon viewing parameters.

The video encoder 130 may transmit (e.g., send, deliver, provide, etc.)the video to a device requesting the video content. The video encoder130 and one or more of the video source 110 and the video preprocessor120 may be integrated into one or more computing devices. The videopreprocessor 120 and the video encoder 130 may be local to one another(e.g., in the same room, on the same premises, etc.). The videopreprocessor 120 and the video encoder 130 may be remote from oneanother.

The video encoder 130 may transmit (e.g., send, deliver, provide, etc.)the encoded video data to the video decoder 140 in a video stream. Thevideo stream may comprise one or more quantization matrices. Thequantization matrices indicated in the video stream may be associatedwith a portion of the video stream. The quantization matrix or matricesindicated in the video stream may have been that/those used to encodethe portion of the video stream. The quantization matrices in the videostream may be indicated by one or more sequence parameter sets (SPS)and/or picture parameter sets (PPS) of the video stream. Thequantization matrices may be transmitted (e.g., sent, delivered,provided, etc.) to the decoder outside of the video stream.

The video decoder 140 may comprise one or more logical blocks ofinstructions 142, 144 for decoding video. The video decoder 140 may berealized as a user device or component thereof. The video decoder 140may receive the encoded video data from the video encoder 130. The videodecoder 140 may receive the determined quantization matrix and/ormatrices associated with the encoded video data. The video decoder 140may receive the quantization matrix and/or matrices as part of the videostream indicating the encoded video data. The video decoder 140 mayexecute one or more of the one or more logical blocks of instructions142, 144 to convert (e.g., decode, determine, etc.) received encodedvideo data to decoded (e.g., uncompressed, determined, etc.) video data.The video decoder 140 may decompress the encoded video data using anentropy decoder. The video decoder 140 may perform an inverse transformon the decompressed quantized transform coefficients from the videostream, thus determining a decoded image representing the originalpartition. The decoded image may be not an exact replication of theoriginal. It may be likely that the decoded image is not an exactreplication of the original. The image data with spatial frequenciesthat were determined to have transform coefficients of zero by thequantization process (e.g., according to the quantization matrix basedon viewing parameters and/or a CSF that was based on the viewingparameters) may be absent from the decoded image. At least some of theabsent image data may be image data with spatial frequencies that arenot perceivable by a viewer under viewing conditions indicated by theone or more viewing parameters.

The encoded video data represented in the video stream sent to the videodecoder 140 may be selected from a plurality of candidate sets ofencoded video data. The candidate sets of encoded video data each may beencoded using a different quantization matrix. The differentquantization matrices each may correspond to (e.g., be based on) aprofile of viewing parameter(s). The encoded video data that is selectedfor transmission to the video decoder 140 associated with particularviewing parameter(s) may have been encoded using a quantization matrixcorresponding to a profile of view parameter(s) that matches or mostclosely matches the actual particular viewing parameter(s) associatedwith the video decoder 140.

The video decoder 140 may output or cause to output the decoded videodata. The decoded video data may be output to a viewer, such as via theuser device and/or a display associated with the user device. Thedecoded video may be output to a viewer under conditions the same orsimilar, at least in some aspects, to the viewing conditions reflectedin the viewing parameter(s) that were used to determine the quantizationmatrix applied during the earlier encoding process.

Any combination or sub-combination of the video source 110, videopreprocessor 120, video encoder 130, and/or video decoder 140 may belocated local (e.g., in the same room, same premises, etc.) to oneanother. The video source 110 may be located at a first location, thevideo encoder 130 (and/or the video preprocessor 120) may be located ata second location, and the video decoder 140 may be located at a thirdlocation, with the first, second, and third locations being differentfrom one another. The video source 110 and the video encoder 130 may belocated at a first location and the video decoder 140 may be located ata second location. Any combination or sub-combination of the videosource 110, video preprocessor 120, video encoder 130, and/or videodecoder 140 may be realized as a computing device, such as a userdevice. The video source 110 and video encoder 130 (and the videopreprocessor 120, if so applicable) may be realized as a first computingdevice (e.g., a first mobile and/or user device, etc.) and the videodecoder 140 may be realized as a second, different computing device(e.g., a second mobile and/or user device, etc.). The video encoder 130and video decoder 140 (and/or the video source 110 and videopreprocessor 120, as the case may be) may be realized as a unitarycomputing device. The decoding performed by the video decoder 140 andthe encoding performed by the video encoder 130 may be part of a videoprocessing operation acting, at least initially, on raw video data. Thevideo processing operation may ultimately output encoded video data byalternately performing encoding and decoding over several iterations.

A video delivery system (e.g., a Video-On-Demand (VOD) system) maycomprise, at least in part, a video distribution system (e.g., the videodistribution system 100). The video delivery system may comprise aserver (e.g., a VOD server), which may be or may comprise a video source(e.g., the video source 110). The server may receive a request for videocontent from a set-top box. The system may comprise a processing unit toreceive raw video from the server. The server may transmit raw videorelated to the received request to the processing unit. The processingunit may be or may comprise a video preprocessor (e.g., the videopreprocessor 120) and/or a video encoder (e.g., the video encoder 130).The processing unit may comprise one or more logical blocks ofinstructions to preprocess received video for easier (e.g., moreefficient, faster, etc.) processing by a video encoder. The processingunit may comprise one or more logical blocks of instructions to encoderaw and/or preprocessed video data. The processing unit may transmit theprocessed (e.g., encoded) video to the set-top box. The set-top box maycomprise a video decoder (e.g., the video decoder 140). The videodecoder may decode the encoded video data. The set-top box may causeoutput of the resultant decoded video, such as via a display associatedwith the set-top box.

FIG. 2 shows a graph 200 relating to contrast sensitivity. The graph 200comprises a vertical axis indicating a contrast sensitivity measurementand a horizontal axis indicating an increasing spatial frequency (e.g.,cycles per degree). Contrast sensitivity may refer to an individual'sability to visually differentiate between—or a sensitivity to—two ormore visual aspects in a video or image. Those visual aspects may relateto luminance (perceived “brightness”), as may be the case when a videoor image comprises contiguous areas having varying degrees of luminance(e.g., black and white areas). Contrast sensitivity to varying degreesof luminance may be referred to as achromatic contrast sensitivity.Contrast sensitivity relating to chrominance (i.e., perceived “color”)may be referred to as chromatic contrast sensitivity. Chromatic contrastsensitivity may indicate an ability to differentiate between contiguousareas having varying chrominance characteristics, such as contiguousareas having two different colors, a color and white, or a color andblack.

Due to limitations of the human eye and visual system, contrastsensitivity to luminance is significantly greater than contrastsensitivity to chrominance. As a consequence, visual information in animage is more significantly carried in the luminance aspects rather thanthe chrominance aspects. Taking an extreme example, video contentdisplayed on a black-and-white television set is stripped of allchrominance aspects, yet may be readily perceived and understood by aviewer. In the converse case, however, video content displayed with onlychrominance components and no luminance would be largely indecipherableto a viewer.

Contrast sensitivity may be individual-specific, although the HumanVisual System (HVS) model may provide a standard measure for contrastsensitivity, as well as other attributes or functions relating to visualperception. Reference to contrast sensitivity and other associatedconcepts may be made with respect to the HVS, unless indicatedotherwise.

Further relating to chrominance, a chromatic contrast sensitivity mayseparately describe a sensitivity to one or more components of color ina color space or system. In a YUV-type color space, Y representsluminance, U represents a first color component, and V represents asecond color component. YCbCr is one type of YUV color space, with Yagain representing a luminance component, Cb representing ablue-difference chroma component (e.g., blue minus Y), and Crrepresenting a red-difference chroma component (e.g., red minus Y). TheYCbCr color space is commonly associated with digital video, as opposedto its YPbPr analog counterpart. Another color space may be ICtCp, withI indicating a luma component, Ct indicating a blue-yellow chromacomponent, and Cp indicating a red-green chroma component. A chromaticcontrast sensitivity may separately indicate a sensitivity to aparticular one of the U (e.g., Cb) or the V (e.g., Cr) color components.A chromatic contrast sensitivity may indicate a sensitivity to acombination of the U and the V color components. A chromatic contrastsensitivity may apply to any color space and/or any combination of colorspaces.

A CSF may represent a relationship between contrast sensitivity and aspatial frequency of an associated visual stimuli. A CSF may indicatethe contrast thresholds at which variations in luminance and/orchrominance, along a continuum of spatial frequencies, can no longer beresolved. A visual stimuli associated with contrast sensitivity (e.g.,achromatic contrast sensitivity, etc.) may comprise a series ofalternating black and white bars arranged as a sine-wave grating. Thewidths of the black and white bars progressively decrease (i.e., theirspatial frequency increases) and the contrast between neighboring blackand white bars likewise decreases progressively along the bars' lengths.A similar sine-wave grating may be configured with bars of alternatingcolors rather than black and white. The alternating colors presented ina sine-wave grating may each be colors presented without a luminancecomponent (“chromaticity”). A sine-wave grating comprising alternatingcolors may be associated with a chromatic contrast sensitivity and thusalso a chromatic CSF. It is already noted that a black and whitesine-wave grating may be associated with an achromatic contrastsensitivity. Thus, a black and white sine-wave grating may beadditionally associated with an achromatic CSF.

The graph 200 comprises a sine-wave grating 204 with a series ofalternating black and white bars. The progressive increase of spatialfrequency towards the right in the graph 200 is visually reflected inthe corresponding decrease in the width of each bar, which also has theeffect of decreasing the distance between successive bars of the sametype (e.g., from black bar to black bar or from white bar to white bar).It will also be observed that the contrast between neighboring black andwhite bars is greatest at the bottom of the graph 200 proximate thehorizontal axis and progressively decreases as the bars extendvertically upward from the horizontal axis. In this respect, thevertical axis further maps a progressively decreasing contrast.

A CSF 202 is shown on the graph 200. The CSF 202 may be determinedaccording to one or more viewing parameters. As already noted, a CSFindicates a contrast threshold at which a person (according to the HVSmodel) is no longer able to perceive variations in contrast over acontinuum of one or more spatial frequencies. Here, the (achromatic) CSF202 indicates the contrast threshold at which a person becomes unable toperceive the transitions between the black and white bars over thecontinuum of spatial frequencies indicated by the horizontal axis. Thus,a person would be unable to perceive the transitions between the whiteand black bars under the contrast and spatial frequency conditionsrepresented by the area(s) of the graph 200 that are above the CSF 202.Conversely, a person would be able to perceive the transitions betweenthe black and white bars under the conditions represented by the area(s)below the CSF 202. The perceptibility of the black and white barsindicated by the CSF 202 is borne out by the visual representation ofthe black and white sine-wave grating 204. While the graph 200 and CSF202 relate to achromatic contrast sensitivity, these concepts may beequally applicable to chromatic contrast sensitivity.

By determining a CSF, based on viewing parameters, and using that CSF togenerate a quantization matrix that is then applied to video data, asignificant reduction of an overall High Dynamic Range (HDR) videobit-rate may be achieved with substantially no degradation in theperceived visual quality. The CSF may be based on a Human Visual System(HVS) nonlinear transformation model followed by a Modulation TransferFunction (MTF). Additionally or alternatively, the CSF may be generated,for example, in accordance with one or more of the following: Barten(Ramp) threshold and Schreiber threshold.

Table 1 shows a first 4×4 quantization matrix.

TABLE 1 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16

The first quantization matrix shown in Table 1 may be a flatquantization matrix. The first quantization matrix shown in Table 1 maybe associated with video content. The first quantization matrix may havebeen determined without use of one or more parameters. The firstquantization matrix may have been determined without use of a viewingparameter associated with playback of the video content. The firstquantization matrix may refer to luma and/or chroma (e.g., one of twochroma components). The first quantization matrix may refer tointer-coding or intra-coding.

Table 2 shows a second 4×4 quantization matrix.

TABLE 2 16 16 16 16 16 16 16 16 16 16 17 18 16 16 18 19

The second quantization matrix shown in Table 2 may be determined basedon one or more variables. The second quantization matrix shown in Table2 may be associated with video content. The second quantization matrixmay be determined based on one or more viewing parameters associatedwith playback of the video content. The second quantization matrix maybe determined based on a CSF. The CSF may be based on one or moreviewing parameters associated with playback of the video content. TheCSF may indicate the spatial frequencies that the HVS is able toperceive under the viewing conditions.

Based on the second quantization matrix shown in Table 2, encoded videodata may be determined that omits at least some image data that the HVSis not able to perceive under the viewing conditions. The secondquantization matrix may comprise values higher (e.g., values of 17, 18,and 19) than the corresponding values (values of 16) found in the flat,first quantization matrix shown in Table 1. These higher values may beobserved in the lower right portion of the second quantization matrix. Alower right portion of a quantization matrix (and a DCT-transformedmatrix, likewise) may generally correspond to higher spatialfrequencies. Accordingly, the 17, 18, and 19 values in the secondquantization matrix may cause the encoding process, based on the secondquantization matrix, to omit the image data at the higher spatialfrequencies corresponding to the 17, 18, and 19 values in the secondquantization matrix. The omitted image data may comprise image data thatis not perceptible by the HVS. Encoded video data determined based onuncompressed video data and the first quantization matrix shown in Table1 may comprise image data that is omitted from encoded video datadetermined based on the same uncompressed video data and the secondquantization matrix (instead of the first). The second quantizationmatrix shown in Table 2 may refer to luma and/or chroma (e.g., one oftwo chroma components). The second quantization matrix may refer tointer-coding or intra-coding.

A quantization matrix, based on the determined CSF, may be customizedsuch that relatively high spatial frequencies, which are notdetectable/visible by the HVS, are removed, thereby allowing reductionof the overall bit-rate of the encoded video data without decreasing thevisual video quality and/or minimizing decreases to visual videoquality. The HDR video may be compressed in a much more efficient mannerby allowing an encoder to allocate more bits to more important and/ornoticeable regions within each video frame and fewer bits to highspatial frequencies regions, which may be less important and/or lessnoticeable regions, instead of allocating equal bits to all regions ofeach video frame.

FIG. 3 is a flow diagram of a method. Video content comprising aplurality of frames may be received at step 310. The video contentand/or the plurality of frames may comprise raw video data, such as HighDynamic Range (HDR) uncompressed video data. The video content may bereceived by a video encoder (e.g., the video encoder 130 in FIG. 1 )and/or a video preprocessor (e.g., the video preprocessor 120 in FIG. 1).

At step 320, one or more of first luminance pixel data associated with afirst frame of the plurality of frames and first chrominance pixel dataassociated with the first frame may be determined, such as by the videoencoder 130 in FIG. 1 and/or the video preprocessor 120 in FIG. 1 . Thefirst frame may comprise a plurality of partitions, and the one or moreof first luminance pixel data and first chrominance pixel data may beassociated with a first partition of the plurality of partitions. Theplurality of partitions may comprise a plurality of at least one ofblocks, macroblocks, and code tree units. The one or more of firstluminance pixel data and first chrominance pixel data may compriseresidual image data. The residual image data may result from aprediction step in an encoding process, such an intra-frame predictionand/or inter-frame prediction. The residual image data may comprisespatial residual data. The spatial residual data may refer to a secondpartition of the first frame. The residual image data may comprisetemporal residual data. The temporal residual data may refer to anotherpartition of a prior or following frame of the plurality of frames.

At step 330, a first viewing parameter associated with playback of thevideo content may be determined. The first viewing parameter associatedwith playback of the video content may be determined by the videoencoder 130 in FIG. 1 and/or the video preprocessor 120 in FIG. 1 . Thefirst viewing parameter may indicate a viewing condition associated withplayback of the video content. The first viewing parameter may compriseone or more of a viewing distance, a pixel density, a pixel length of asinusoidal grating cycle, a width of a display associated with playbackof the video content, a viewing angle, ambient illumination level, and areflection coefficient of a display associated with playback of thevideo content.

A viewing parameter may be determined by real-time or near real-timefeedback. The viewing parameter may be determined using a sensor, whichmay be positioned proximate the viewing environment. The sensor may beassociated with or form part of the decoder or other device (e.g., aset-top device) causing output of the video content. The sensor may beincorporated with a remote control device, such as to measure and/orestimate a relative distance between a viewer and the display. Themeasured viewing parameter may be communicated to the video encoder 130in FIG. 1 and/or video preprocessor immediately or according to a longertime interval.

A viewing parameter may be pre-determined at the time of encoding thevideo content. The video encoder 130 in FIG. 1 and/or an entityassociated with the video encoder 130 in FIG. 1 may already store theviewing parameter when the video content is encoded. Such viewingparameter may be based on information previously provided by anexpectant viewer, a user/entity associated with the configuration of theviewing environment, a user/entity providing the video stream, etc. Acable provider may be aware of a room type in which a set-top box isinstalled and one or more viewing parameters may be determined orestimated according to that room type. A technician may measure one ormore viewing parameters during an installation of the set-top box. Thetechnician and/or the viewer may indicate the make and model of thedisplay associated with the set-top box, from which various viewingparameters may be determined (e.g., display resolution, etc.).

At step 340, a first quantization matrix associated with the first framemay be generated (e.g., determined) based on the first viewingparameter. The first quantization matrix may be additionally oralternatively based on the first frame. The first quantization matrixassociated with the first frame may be additionally or alternativelybased on the one or more of first luminance pixel data and firstchrominance pixel data associated with the first frame. The videoencoder 130 in FIG. 1 and/or the video preprocessor 120 in FIG. 1 maydetermine the first quantization matrix.

The generating the first quantization matrix may comprise adjusting(e.g., modifying, updating, etc.), at least in part, apreviously-determined quantization matrix associated with the videocontent. Adjusting the previously-determined quantization matrix maycomprise determining the first quantization matrix based on thepreviously-determined quantization matrix. Adjusting thepreviously-determined quantization matrix to determine the firstquantization matrix may be based on a corresponding change in viewingparameters (e.g., an increase or decrease to an ambient illuminationlevel in the viewing environment). The previously-determinedquantization matrix may be one determined for another (e.g., previous)partition of the first frame or for another (e.g., previous) frame (orpartition thereof) of the plurality of frames.

Adjusting, at least in part, the previously-determined quantizationmatrix may be performed on a matrix entry-by-matrix entry basis. Forexample, determining a first entry of the first quantization matrix maycomprise adjusting (e.g., modifying or updating) a corresponding firstentry of the previously-determined quantization matrix. As anotherexample, determining a first entry of the first quantization matrix maycomprise carrying forward (i.e., without changing) or copying acorresponding first entry of the previously-determined quantizationmatrix to the first entry of the first quantization matrix.

The first quantization matrix may be determined based on a firstcontrast sensitivity function (CSF). The first CSF may be determined(e.g., generated) based on the first viewing parameter. The first CSFmay be determined based, additionally or alternatively, on one or moreof the first frame and the one or more of first luminance pixel data andfirst chrominance pixel data. The video encoder 130 in FIG. 1 maydetermine the first CSF. The video preprocessor 120 in FIG. 1 maydetermine the first CSF.

The first quantization matrix and/or the first CSF may be determinedbased on a characteristic of the first partition and/or the first framecomprising the first partition. The first quantization matrix and/or thefirst CSF may be determined based on the first frame comprising a scenecut.

At least a first portion of a coded video stream may be determined(e.g., generated) based on the first frame and the first quantizationmatrix. The video encoder 130 in FIG. 1 may determine at least a firstportion of a coded video stream based on the first frame and the firstquantization matrix. The video preprocessor 120 in FIG. 1 may determineat least a first portion of a coded video stream based on the firstframe and the first quantization matrix.

Determining (e.g., generating) at least the first portion of the codedvideo stream may comprise determining, based on a linear transformationof at least a portion of the one or more of first luminance pixel dataand first chrominance pixel data, a plurality of transform coefficientsassociated with the first frame. The video encoder 130 and/or the videopreprocessor 120 in FIG. 1 may determine the plurality of transformcoefficients associated with the first frame. The plurality of transformcoefficients may be organized in a table or matrix corresponding to thefirst quantization matrix. Determining at least the first portion of thecoded video stream may comprise quantizing, based on the firstquantization matrix, the plurality of transform coefficients. The videoencoder 130 and/or video preprocessor 120 in FIG. 1 may quantize theplurality of transform coefficients based on the first quantizationmatrix. Determining at least the first portion of the coded video streammay comprise performing entropy encoding on the quantized plurality oftransform coefficients. The video encoder 130 and/or video preprocessor120 in FIG. 1 may perform the entropy encoding on the quantizedplurality of transform coefficients.

One or more of second luminance pixel data associated with a secondframe of the plurality of frames and second chrominance pixel dataassociated with the second frame may be determined, such as by the videoencoder 130 and/or video preprocessor 120 in FIG. 1 . A second viewingparameter associated with playback of the video content may bedetermined, such as by the video encoder 130 and/or video preprocessor120 in FIG. 1 . A second quantization matrix associated with the secondframe may be determined based the second viewing parameter. The secondquantization matrix associated with the second frame may be based,additionally or alternatively, on one or more of the second frame andthe one or more of second luminance pixel data and second chrominancepixel data. The video encoder 130 and/or the video preprocessor 120 inFIG. 1 may determine the second quantization matrix associated with thesecond frame.

At least a second portion of the coded video stream may be determined(e.g., generated) based on the second frame and the second quantizationmatrix. The video encoder 130 and/or the video preprocessor 120 in FIG.1 may determine the at least a second portion of the coded video streambased on the second frame and the second quantization matrix. A secondCSF may be determined based on the second viewing parameter. The secondCSF may be based, additionally or alternatively, on one or more of thesecond frame and the one or more of second luminance pixel data andsecond chrominance pixel data. The video encoder 130 and/or the videopreprocessor 120 in FIG. 1 may determine the second CSF. The secondquantization matrix may be based, additionally or alternatively, on thesecond CSF.

The first quantization matrix may be transmitted (e.g., sent, delivered,provided, etc.) to an associated decoder. The video encoder 130 and/orthe video preprocessor 120 in FIG. 1 may transmit the first quantizationmatrix to the video decoder 140 in FIG. 1 . The first quantizationmatrix may be transmitted to the decoder in conjunction with theassociated coded video stream. The first quantization matrix may betransmitted to the decoder as part of the coded video stream. Thedecoder may use the first quantization matrix in decoding the encodedvideo stream.

A user may cause a set-top box to request video content from a videodelivery system, such as a video on-demand system. The set-top box maybe configured to decode encoded video content, such as indicated in anencoded video stream. The set-top box may be configured to determine aviewing parameter. The set-top box may comprise a sensor, such as asensor for measuring the level of ambient light in the room with theset-top box. The set-top box may transmit (e.g., send, deliver, provide,etc.) a signal indicative of the sensed level of ambient light to avideo encoder or other system associated with the video delivery system.The video encoder (or other system) may determine a quantization matrixbased on the sensed ambient lighting in the set-top box's room or otherviewing environment. The video encoder (or other system) may determine aCSF based on the ambient lighting in the set-top box's room. Thequantization matrix may be determined based on the CSF. The videoencoder may receive the video content in a raw video format. The videoencoder may encode the requested video content based on the quantizationmatrix. Encoding the video content may comprise determining, based onthe quantization matrix, a quantized transform (e.g., DCT) coefficientmatrix. The quantized transform coefficient matrix may be subjected toentropy encoding. The encoded video content may comprise theentropy-encoded transform coefficient matrix.

With the encoded video content being based on the quantization matrix(which may be based in turn on the level of ambient lighting) theencoded video content may be customized to omit image data in theencoded video content that would be imperceptible to the user underthose ambient lighting conditions in the set-top box's room. The encodedvideo content may therefore require less bandwidth than it wouldotherwise. The encoded video content may comprise video content thatunderwent enhanced encoding from processing resources that were divertedfrom the processing resources that otherwise were intended to encode theomitted image data. The encoded video content may comprise video contentrepresented by additional bits of image data that would otherwise haverepresented the omitted image data. Thus the encoded video content maybe transmitted at a reduced bandwidth, with better image quality, orsome combination thereof.

The video encoder may transmit (e.g., send, deliver, provide, etc.) theencoded video content to the set-top box via a network. The quantizationmatrix may be transmitted to the set-top box via the network. Theencoded video content may be transmitted to the decoder as an encodedvideo stream comprising a plurality of encoded frames. The set-top boxmay decode the encoded video content using the one or more quantizationmatrices. The set-top box may cause the decoded video content to beoutput via an associated display device, such as a television.

Encoded video content, which may be indicated via an encoded videostream, may be transmitted to a video decoder. A quantization matrixassociated with at least a portion of the encoded video content, may betransmitted to the video decoder. The encoded video content may comprisethe quantization matrix. The decoder may decode at least a portion ofthe encoded video content based on the associated quantization matrix.

FIG. 4 is a flow diagram of a method. Video content comprising aplurality of coded frames may be received, at step 410. A video decoder(e.g., the video decoder 140 in FIG. 1 ) may receive the video contentcomprising the plurality of coded frames. The video content may compriseHigh Dynamic Range (HDR) compressed video content, but is not solimited. The video content may comprise Standard Dynamic Range (SDR)compressed video content.

At step 420, one or more of first luminance pixel data associated with afirst coded frame of the plurality of coded frames and first chrominancepixel data associated with the first coded frame may be determined. Thevideo decoder 140 in FIG. 1 may determine the one or more of firstluminance pixel data associated with a first coded frame of theplurality of coded frames and first chrominance pixel data associatedwith the first coded frame. The one or more of first luminance pixeldata and first chrominance pixel data may comprise a plurality ofquantized transform coefficients. The first coded frame may comprise aplurality of partitions and the one or more of first luminance pixeldata and first chrominance pixel data may be associated with a firstpartition of the plurality of partitions. The plurality of partitionsmay comprise a plurality of at least one of blocks, macroblocks, andcoding tree units. The one or more of first luminance pixel data andfirst chrominance pixel data may comprise residual image data. Theresidual image data may comprise spatial residual data. The residualimage data may comprise temporal residual data.

At step 430, a first quantization matrix may be received. The firstquantization matrix may have been determined based on (e.g., derivedfrom) a first viewing parameter associated with playback of the videocontent. The first quantization matrix may be associated with the firstcoded frame of the plurality of coded frames. The first quantizationmatrix may be transmitted and/or received in association withtransmitting and/or receiving the first coded frame. The first codedframe may have been determined (e.g., encoded, compressed, etc.) basedon the first quantization matrix. The video decoder 140 in FIG. 1 mayreceive the first quantization matrix. The first viewing parameter maycomprise one or more of a viewing distance, a pixel density, a pixellength of a sinusoidal grating cycle, a width of a display associatedwith playback of the video content, a viewing angle, ambientillumination level, and a reflection coefficient of a display associatedwith playback of the video content.

The first viewing parameter may be associated with the decoder and/orthe device requesting the video content (which may be one and the same).The decoder and/or other requesting device may be the device via whichthe decoded video content is expected to be output to the viewer. Thedecoder and/or other requesting device may comprise the display viawhich the video content is expected to be output to the viewer. Thefirst viewing parameter may reflect a viewing condition present at theenvironment or location of the decoder and/or other device requestingthe video content. The first view parameter may have been transmittedfrom the decoder and/or the requesting device to the video encoder, thevideo preprocessor, and/or another associated entity. The first viewingparameter may have been indicated by a user, owner, and/or servicingentity associated with the decoder and/or requesting device. The firstviewing parameter may have been indicated by a user, owner, and/orservicing entity before the video content was requested.

A set-top box may comprise the decoder and may be situated in a livingroom. The set-top box may be connected to a television display. Theset-top box may request the video content. The viewing parameter used todetermine the first quantization matrix may indicate a characteristic ofthe living room (e.g., the ambient lighting in the living room, thedistance between the television display and furniture, the viewing anglebetween the television display and the furniture, etc.) and/or thetelevision display (e.g., the dimensions of the display, the re solutionof the display, etc.).

At step 440, decoded video data may be generated (e.g., determined)based on the one or more of first luminance pixel data and firstchrominance pixel data and the first quantization matrix. The decodedvideo data may be output (e.g., caused to be output) based on the one ormore of first luminance pixel data and first chrominance pixel data. Thedecoded video data may comprise a decoded first frame (corresponding tothe first frame) of a plurality of decoded frames. The video decoder 140in FIG. 1 may cause output of decoded video data based on the one ormore of first luminance pixel data and first chrominance pixel dataassociated with the first frame and the first quantization matrix.Generating (e.g., determining) the decoded video data may comprisedetermining, based on the first quantization matrix and the plurality ofquantized transform coefficients (e.g., indicated by or in the one ormore of first luminance pixel data and first chrominance pixel data), aplurality of rescaled transform coefficients. The video decoder 140 inFIG. 1 may determine the plurality of rescaled transform coefficients.Generating the decoded video data may comprise performing an inverselinear transformation of the plurality of rescaled transformcoefficients. The video decoder 140 in FIG. 1 may perform the inverselinear transformation of the plurality of rescaled transformcoefficients.

The decoded video data may be based on the inverse-transformed pluralityof rescaled transform coefficients. A plurality of rescaled transformcoefficients associated with a first partition of the first coded framemay be determined based on the first quantization matrix and the one ormore of first luminance pixel data and first chrominance pixel data. Thevideo decoder 140 in FIG. 1 may determine the plurality of rescaledtransform coefficients associated with the first partition. Causingoutput of the decoded video data may be further based on the pluralityof rescaled transform coefficients associated with the first partitionof the first coded frame.

Causing output of the decoded video data may comprise causing thedecoded video data to be output via a display. Causing the decoded videodata to be output via the display may comprise transmitting a signal,indicating the decoded video data, to the display. The display maycomprise a television display communicatively connected to the decoder(e.g., a set-top box). The display may comprise a display on a mobiledevice and the mobile device may comprise the decoder. The display maycomprise a computer display and the computer may comprise the decoder.

One or more of second luminance pixel data associated with a secondcoded frame of the plurality of coded frames and second chrominancepixel data associated with the second coded frame may be determined. Thevideo decoder 140 in FIG. 1 may determine one or more of secondluminance pixel data associated with the second coded frame of theplurality of coded frames and second chrominance pixel data associatedwith the second coded frame. A second quantization matrix, based on(e.g., derived from) a second viewing parameter associated with playbackof the video content, may be received. The video decoder 140 in FIG. 1may receive the second quantization matrix. The decoded video data maybe further based on the one or more of second luminance pixel data andsecond chrominance pixel data and the second quantization matrix.

FIG. 5 is a flow diagram of a method. Uncompressed video data comprisinga plurality of frames may be received, at step 510. The video encoder130 in FIG. 1 may receive the uncompressed video data comprising aplurality of frames. The video preprocessor 120 in FIG. 1 may receivethe uncompressed video data comprising a plurality of frames. A frame ofthe plurality of frames may comprise a plurality of partitions. Apartition of the plurality of partitions may indicate a plurality ofimage values. A partition may comprise a block, a macroblock, and/or acoding tree unit. An image value of a partition may comprise a pixelvalue. Thus, a partition may comprise a plurality of pixel values. Theplurality of pixel values may comprise an 8×8 set of image values. Animage value may indicate a luminance value and/or a chrominance value(e.g., a U or V value).

At step 520, a first plurality of image values of a first partition of afirst frame of the plurality of frames may be determined. The videoencoder 130 in FIG. 1 may determine the first plurality of image valuesof the first partition of the first frame of the plurality of frames.The video preprocessor 120 in FIG. 1 may determine the first pluralityof image values of the first partition of the first frame of theplurality of frames. The first plurality of image values may indicate atleast one of luminance pixel values and chrominance pixel values. Thefirst plurality of image values may indicate residual values, such asspatial residual values or temporal residual values. The first partitionmay comprise at least one of a block, a macroblock, and a coding treeunit.

At step 530, a first viewing parameter associated with playback of thevideo data may be determined. The video encoder 130 in FIG. 1 maydetermine the first viewing parameter associated with playback of thevideo data. The video preprocessor 120 in FIG. 1 may determine the firstviewing parameter associated with playback of the video data. The firstviewing parameter may comprise one or more of a viewing distance, apixel density, a pixel length of a sinusoidal grating cycle, a width ofa display associated with playback of the video data, a viewing angle,ambient illumination level, and a reflection coefficient of a displayassociated with playback of the video data.

At step 540, a first contrast sensitivity function (CSF) may bedetermined based on the first viewing parameter. The first CSF may bedetermined based on, additionally or alternatively, the first pluralityof image values of the first partition. The video encoder 130 in FIG. 1may determine the first CSF. The video preprocessor 120 in FIG. 1 maydetermine the first CSF. The first CSF may reflect the expected viewer'sability to perceive differences in image detail and/or motion under thedetermined viewing parameters. The first CSF may indicate generally thatthe viewer will be able to perceive greater image detail and/or motionthan would otherwise. The first CSF may indicate generally that theviewer will have a reduced ability to perceive image detail and/ormotion. The first CSF may indicate that the viewer will have a greaterability to perceive image detail and/or motion at some subset of thespectrum of spatial frequencies while the viewer will have reducedability to perceive image detail and/or motion at another, differentsubset of the spectrum of spatial frequencies.

The determining the first CSF may comprise modifying (e.g., adjusting,updating, etc.) a standardized CSF that is based on the HVS but isagnostic as to any other factors that may affect a person's (e.g., aviewer's) ability to differentiate image details and/or motions beyond aspatial frequency threshold. The standardized CSF may indicate that aviewer should be able to perceive certain image details and/or motion,but, in fact, the viewer is not able to perceive those image detailsand/or motion under the actual viewing parameters. The first CSF mayaccount for the effects, positive or negative, to image detail and/ormotion perception that the viewing parameters are expected to cause.

At step 550, a first quantization matrix may be determined based on thefirst CSF. The video encoder 130 in FIG. 1 may determine the firstquantization matrix based on the first CSF. The video preprocessor 120in FIG. 1 may determine the first quantization matrix based on the firstCSF. The first CSF may indicate the spatial frequency threshold beyondwhich a viewer, under the viewing parameters, will be unable toperceived differences in image detail and/or motion. The firstquantization matrix may be determined such as to omit image data thatwill be or is likely to be imperceptible to a viewer under thedetermined viewing parameters (e.g., the viewing conditions duringplayback, aspects of the display, etc.).

At step 560, a corresponding first plurality of transform coefficients(associated with the first partition) may be determined based on adiscrete cosine transform (DCT) of the first plurality of image valuesof the first partition. The video encoder 130 in FIG. 1 may determinethe corresponding first plurality of transform coefficients based on aDCT of the first plurality of image values of the first partition. Thevideo preprocessor 120 in FIG. 1 may determine the corresponding firstplurality of transform coefficients based on a DCT of the firstplurality of image values of the first partition. The first plurality oftransform coefficients may be formed as a matrix of transformcoefficients.

At step 570, the first plurality of transform coefficients may bequantized based on the first quantization matrix. The video encoder 130in FIG. 1 may quantize the first plurality of transform coefficientsbased on the first quantization matrix. The video preprocessor 120 inFIG. 1 may quantize the first plurality of transform coefficients basedon the first quantization matrix. The first plurality of transformcoefficients may be quantized such as to omit image data that would beimperceptible to the viewer under the determined viewing parameters. Thetransform coefficients that correspond to the image data that aredetermined to be imperceptible to the viewer may be reduced to “0”within the plurality of transform coefficients.

At step 580, the quantized first plurality of transform coefficients maybe output. The video encoder 130 in FIG. 1 may output the quantizedfirst plurality of transform coefficients. The video preprocessor 120 inFIG. 1 may output the quantized first plurality of transformcoefficients. The outputting the first plurality of transformcoefficients may comprise entropy encoding the quantized first pluralityof transform coefficients. The video encoder 130 in FIG. 1 may entropyencode the quantized first plurality of transform coefficients. Thevideo preprocessor 120 in FIG. 1 may entropy encode the quantized firstplurality of transform coefficients. The entropy encoding may comprisefixed length coding, variable length coding, context adaptive variablelength coding, and/or context adaptive binary arithmetic coding. Theentropy encoding may comprise Huffman encoding.

The outputting the first plurality of transform coefficients maycomprise outputting compressed video data. The video encoder 130 in FIG.1 may output the compressed video data. The video preprocessor 120 inFIG. 1 may output the compressed video data. The compressed video datamay comprise the entropy-encoded, quantized first plurality of transformcoefficients. The compressed video data may be incorporated in anencoded video stream. The compressed video data may be output forplayback. A decoder may receive the compressed video data (e.g., theencoded video stream) and decode the compressed video data. The decodedvideo data may be output (e.g., presented) to a viewer, such as via adisplay device.

For one or more additional partitions of the plurality of partitions ofthe first frame, a corresponding plurality of transform coefficients ofan additional partition may be determined based on a discrete cosinetransform (DCT) of the plurality of image values of the additionalpartition. The video encoder 130 in FIG. 1 may determine thecorresponding plurality of transform coefficients of the additionalpartition. The video preprocessor 120 in FIG. 1 may determine thecorresponding plurality of transform coefficients of the additionalpartition. The plurality of transform coefficients of the additionalpartition may be quantized based on the first quantization matrix. Thevideo encoder 130 in FIG. 1 may quantize the plurality of transformcoefficients of the additional partition based on the first quantizationmatrix. The video preprocessor 120 in FIG. 1 may quantize the pluralityof transform coefficients of the additional partition based on the firstquantization matrix.

The quantized plurality of transform coefficients of the additionalpartition may be entropy encoded. The video encoder 130 in FIG. 1 mayentropy encode the quantized plurality of transform coefficients of theadditional partition. The video preprocessor 120 in FIG. 1 may entropyencode the quantized plurality of transform coefficients of theadditional partition. The compressed video data may further comprise theentropy-encoded, quantized plurality of transform coefficients of eachadditional partition of the plurality of partitions of the first frame.

FIG. 6 shows an operating environment 600, but is not intended tosuggest any limitation as to the scope of use or functionality ofoperating environment architecture. Neither should the operatingenvironment be interpreted as having any dependency or requirementrelating to any one or combination of components shown in the operatingenvironment.

The present methods, systems, and apparatuses may be operational withnumerous other general purpose or special purpose computing systemenvironments or configurations. Examples of well-known computingsystems, environments, and/or configurations that may be suitable foruse with the methods, systems, and apparatuses comprise, but are notlimited to, personal computers, server computers, laptop devices, andmultiprocessor systems. Additional examples comprise set top boxes,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, distributed computing environments that comprise any of theabove systems or devices, and the like.

The processing of the disclosed methods, systems, and apparatuses may beperformed by software components. The disclosed methods, systems, andapparatuses may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by one or more computers or other devices. Generally, programmodules comprise computer code, routines, programs, objects, components,data structures, etc. that performs particular tasks or implementsparticular abstract data types. The disclosed methods may be practicedin grid-based and distributed computing environments where tasks may beperformed by remote processing devices that are linked through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote computer storage mediaincluding memory storage devices.

Further, the methods, systems, and apparatuses disclosed may beimplemented via a general-purpose computing device in the form of acomputing device 601. The components of the computing device 601 maycomprise, but are not limited to, one or more processors or processingunits 603, a system memory 612, and a system bus 613 that couplesvarious system components including the processor 603 to the systemmemory 612. In the case of multiple processing units 603, the system mayutilize parallel computing.

The system bus 613 represents one or more of several possible types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, sucharchitectures may comprise an Industry Standard Architecture (ISA) bus,a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, aVideo Electronics Standards Association (VESA) local bus, an AcceleratedGraphics Port (AGP) bus, and a Peripheral Component Interconnects (PCI),a PCI-Express bus, a Personal Computer Memory Card Industry Association(PCMCIA), Universal Serial Bus (USB) and the like. The system bus 613,and all buses specified in this description may be implemented over awired or wireless network connection and each of the subsystems,including the processor 603, a mass storage device 604, an operatingsystem 605, video processing software 606, video processing data 607, anetwork adapter 608, system memory 612, an Input/Output Interface 610, adisplay adapter 609, a display device 611, and a human machine interface602, may be contained within one or more remote computing devices 614a,b,c at physically separate locations, connected through buses of thisform, in effect implementing a fully distributed system.

The computing device 601 typically comprises a variety of computerreadable media. Example readable media may be any available media thatis accessible by the computing device 601 and comprises, for example andnot meant to be limiting, both volatile and non-volatile media,removable and non-removable media. The system memory 612 comprisescomputer readable media in the form of volatile memory, such as randomaccess memory (RAM), and/or non-volatile memory, such as read onlymemory (ROM). The system memory 612 typically contains data such asvideo processing data 607 and/or program modules such as operatingsystem 605 and video processing software 606 that are immediatelyaccessible to and/or are presently operated on by the processing unit603.

The computing device 601 may comprise other removable/non-removable,volatile/non-volatile computer storage media. By way of example, FIG. 6shows a mass storage device 604 which may provide non-volatile storageof computer code, computer readable instructions, data structures,program modules, and other data for the computing device 601. Forexample and not limitation, a mass storage device 604 may be a harddisk, a removable magnetic disk, a removable optical disk, magneticcassettes or other magnetic storage devices, flash memory cards, CD-ROM,digital versatile disks (DVD) or other optical storage, random accessmemories (RAM), read only memories (ROM), electrically erasableprogrammable read-only memory (EEPROM), and the like.

Optionally, any number of program modules may be stored on the massstorage device 604, including by way of example, an operating system 605and video processing software 606. Each of the operating system 605 andvideo processing software 606 (or some combination thereof) may compriseelements of the programming and the video processing software 606. Videoprocessing data 607 may be stored on the mass storage device 604. Videoprocessing data 607 may be stored in any of one or more databases knownin the art. Examples of such databases comprise, DB2®, Microsoft®Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, and the like.The databases may be centralized or distributed across multiple systems.

The user may enter commands and information into the computing device601 via an input device (not shown). Examples of such input devices maycomprise, but are not limited to, a keyboard, pointing device (e.g., a“mouse”), a microphone, a joystick, a scanner, tactile input devicessuch as gloves, and other body coverings, and the like These and otherinput devices may be connected to the processing unit 603 via a humanmachine interface 602 that is coupled to the system bus 613, but may beconnected by other interface and bus structures, such as a parallelport, game port, an IEEE 1394 Port (also known as a Firewire port), aserial port, or a universal serial bus (USB).

A display device 611 may be connected to the system bus 613 via aninterface, such as a display adapter 609. It is contemplated that thecomputing device 601 may have more than one display adapter 609 and thecomputing device 601 may have more than one display device 611. Forexample, a display device may comprise a monitor, an LCD (Liquid CrystalDisplay), or a projector. In addition to the display device 611, otheroutput peripheral devices may comprise components such as speakers (notshown) and a printer (not shown) which may be connected to the computingdevice 601 via Input/Output Interface 610. Any step and/or result of themethods may be output in any form to an output device. Such output maycomprise any form of visual representation, including, but not limitedto, textual, graphical, animation, audio, tactile, and the like. Thedisplay device 611 and computing device 601 may comprise part of onedevice, or separate devices.

The computing device 601 may operate in a networked environment usinglogical connections to one or more remote computing devices 614 a,b,c.By way of example, a remote computing device may comprise a personalcomputer, portable computer, a smart phone, a server, a router, anetwork computer, a peer device or other common network node. Logicalconnections between the computing device 601 and a remote computingdevice 614 a,b,c may be made via a network 615, such as a local areanetwork (LAN) and a general wide area network (WAN). Such networkconnections may be through a network adapter 608. A network adapter 608may be implemented in both wired and wireless environments. Suchnetworking environments are conventional and commonplace in dwellings,offices, enterprise-wide computer networks, intranets, and the Internet.

For purposes of illustration, application programs and other executableprogram components such as the operating system 605 are shown herein asdiscrete blocks, although such programs and components may reside atvarious times in different storage components of the computing device601, and may be executed by the data processor(s) of the computer. Animplementation of video processing software 606 may be stored on ortransmitted across some form of computer readable media. Any of thedisclosed methods may be performed by computer readable instructionsembodied on computer readable media. Computer readable media maycomprise any available media that may be accessed by a computer. By wayof example and not limitation, computer readable media may comprise“computer storage media” and “communications media.” “Computer storagemedia” comprise volatile and non-volatile, removable and non-removablemedia implemented in any methods or technology for storage ofinformation such as computer readable instructions, data structures,program modules, or other data. Example computer storage media maycomprise RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which may be used to store thedesired information and which may be accessed by a computer.

1. A method, comprising: receiving video content; receiving, from a device configured for output of video, a viewing parameter associated with output of the video content; generating, based on the viewing parameter and pixel data associated with the video content, a quantization matrix associated with the video content; and encoding, based at least in part on the quantization matrix, at least a portion of the video content.
 2. The method of claim 1, wherein the pixel data comprises at least one of chrominance pixel data or luminance pixel data.
 3. The method of claim 2, wherein the video content comprises a plurality of frames, further comprising determining one or more of the luminance pixel data associated with a frame of the plurality of frames or the chrominance pixel data associated with the frame.
 4. The method of claim 3, wherein the quantization matrix is further associated with the frame of the plurality of frames, and wherein the generating the quantization matrix comprises determining, based on the viewing parameter and the one or more of the luminance pixel data or the chrominance pixel data, the quantization matrix.
 5. The method of claim 3, wherein the frame comprises a plurality of partitions, and wherein the one or more of the luminance pixel data or the chrominance pixel data is associated with a partition of the plurality of partitions.
 6. The method of claim 2, wherein the encoding the at least a portion of the video content comprises quantizing, based on the quantization matrix, one or more transform coefficients associated with at least a portion of the one or more of the luminance pixel data or the chrominance pixel data, and entropy encoding the quantized one or more transform coefficients.
 7. The method of claim 2, wherein the one or more of the luminance pixel data or the chrominance pixel data comprises residual image data.
 8. The method of claim 1, further comprising determining, based on the viewing parameter and the pixel data, a contrast sensitivity function (CSF), wherein the generating the quantization matrix comprises determining, based on the viewing parameter, the pixel data and the CSF, the quantization matrix.
 9. A device, comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the device to: receive video content; receive, from another device configured for output of video, a viewing parameter associated with output of the video content; generate, based on the viewing parameter and pixel data associated with the video content, a quantization matrix associated with the video content; and encode, based at least in part on the quantization matrix, at least a portion of the video content.
 10. The device of claim 9, wherein the pixel data comprises at least one of chrominance pixel data or luminance pixel data.
 11. The device of claim 10, wherein the video content comprises a plurality of frames, and wherein the instructions, when executed by the one or more processors, further cause the device to determine one or more of the luminance pixel data associated with a frame of the plurality of frames or the chrominance pixel data associated with the frame.
 12. The device of claim 11, wherein the quantization matrix is further associated with the frame of the plurality of frames, and wherein the instructions that cause the device to generate the quantization matrix, when executed by the one or more processors, further cause the device to determine, based on the viewing parameter and the one or more of the luminance pixel data or the chrominance pixel data, the quantization matrix.
 13. The device of claim 11, wherein the frame comprises a plurality of partitions, and wherein the one or more of the luminance pixel data or the chrominance pixel data is associated with a partition of the plurality of partitions.
 14. The device of claim 10, wherein the instructions that cause the device to encode the at least a portion of the video content, when executed by the one or more processors, further cause the device to: quantize, based on the quantization matrix, one or more transform coefficients associated with at least a portion of the one or more of the luminance pixel data or the chrominance pixel data, and entropy encode the quantized one or more transform coefficients.
 15. The device of claim 10, wherein the one or more of the luminance pixel data or the chrominance pixel data comprises residual image data.
 16. The device of claim 9, wherein the instructions, when executed by the one or more processors, further cause the device to determine, based on the viewing parameter and the pixel data, a contrast sensitivity function (CSF), and wherein the instructions that cause the device to generate the quantization matrix, when executed by the one or more processors, further cause the device to determine, based on the viewing parameter, the pixel data and the CSF, the quantization matrix.
 17. A non-transitory computer-readable medium storing instructions that, when executed, cause: receiving video content; receiving, from a device configured for output of video, a viewing parameter associated with output of the video content; generating, based on the viewing parameter and pixel data associated with the video content, a quantization matrix associated with the video content; and encoding, based at least in part on the quantization matrix, at least a portion of the video content.
 18. The non-transitory computer-readable medium of claim 17, wherein the pixel data comprises at least one of chrominance pixel data or luminance pixel data.
 19. The non-transitory computer-readable medium of claim 18, wherein the video content comprises a plurality of frames, wherein the instructions, when executed, further cause determining one or more of the luminance pixel data associated with a frame of the plurality of frames or the chrominance pixel data associated with the frame.
 20. The non-transitory computer-readable medium of claim 19, wherein the quantization matrix is further associated with the frame of the plurality of frames, and wherein the instructions that cause the generating the quantization matrix, when executed, further cause determining, based on the viewing parameter and the one or more of the luminance pixel data or the chrominance pixel data, the quantization matrix.
 21. The non-transitory computer-readable medium of claim 19, wherein the frame comprises a plurality of partitions, and wherein the one or more of the luminance pixel data or the chrominance pixel data is associated with a partition of the plurality of partitions.
 22. The non-transitory computer-readable medium of claim 18, wherein the instructions that cause encoding the at least a portion of the video content, when executed, further cause: quantizing, based on the quantization matrix, one or more transform coefficients associated with at least a portion of the one or more of the luminance pixel data or the chrominance pixel data, and entropy encoding the quantized one or more transform coefficients.
 23. The non-transitory computer-readable medium of claim 18, wherein the one or more of the luminance pixel data or the chrominance pixel data comprises residual image data.
 24. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed, further cause determining, based on the viewing parameter and the pixel data, a contrast sensitivity function (CSF), and wherein the instructions that cause generating the quantization matrix, when executed, further cause determining, based on the viewing parameter, the pixel data and the CSF, the quantization matrix. 